Auditory Encoding of Visual Temporal Sequences

When the senses deliver conflicting information, vision dominates spatial processing, and audition dominates temporal processing. We asked whether this sensory specialization results in cross-modal encoding of unisensory input into the task-appropriate modality. Specifically, we investigated whether visually portrayed temporal structure receives automatic, obligatory encoding in the auditory domain. In three experiments, observers judgedwhether the changes in two successive visual sequences followed the same or different rhythms. We assessed temporal representations by measuring the extent to which both task-irrelevant auditory information and task-irrelevant visual information interfered with rhythm discrimination. Incongruent auditory information significantly disrupted task performance, particularly when presented during encoding; by contrast, varying the nature of the rhythm-depicting visual changes had minimal impact on performance. Evidently, the perceptual system automatically and obligatorily abstracts temporal structure from its visual form and represents this structure using an auditory code, resulting in the experience of ‘‘hearing visual rhythms.’’ People’s interpretation of the world depends on information delivered through multiple senses. Over the past 40 years, numerous studies have investigated how human perceptual systems coordinate this varied input to create a unified description of reality. Early work suggested that vision predominates in multisensory processing: When visual information conflicts with information from other sensory modalities, vision typically ‘‘wins’’ (e.g., Hay, Pick, & Ikeda, 1965; Howard & Templeton, 1966). Recent work, however, converges on a more balanced view: Although vision dominates audition for processing spatial information, audition often dominates vision for processing temporal information (e.g., Bertelson & Aschersleben, 1998; Kitagawa & Ichihara, 2002; Kitajima & Yamashita, 1999; Recanzone, 2003; Repp & Penel, 2002; Wada, Kitagawa, & Noguchi, 2003). For example, a repetitive sound (auditory flutter) presented simultaneously with a flickering light causes the rate of perceived visual flicker to shift toward the auditory flutter rate (e.g., Shipley, 1964; Welch, DuttonHurt, & Warren, 1986). Known as auditory driving, this tendency for visual flicker to become perceptually synchronized with auditory flutter occurs even though the flutter and flicker rates are easily distinguished when presented in succession (Recanzone, 2003). Auditory dominance of temporal perception also shows up in the most simple of experiences: A single flash and single audible click occurring in close temporal proximity shift perceptually toward temporal coincidence, with misperception of the visual stimulus accounting for the bulk of the shift (Fendrich & Corballis, 2001). The conditions yielding dominance of vision versus dominance of hearing likely reflect divergent specializations of visual and auditory processing, respectively. According to the modality-appropriateness hypothesis (Welch, 1999; Welch & Warren, 1980), perception gives precedence to the ‘‘best’’ sensory modality for the task at hand: vision for spatial judgments and audition for temporal judgments. Intersensory conflicts are resolved through subjugation of the less reliable sense—as implied by auditory driving—and possibly even through sensory recalibration (Kitagawa & Ichihara, 2002; Recanzone, 1998, 2003). Largely ignored, however, has been another potential consequence of sensory specialization: cross-modal encoding of unisensory input into the ‘‘appropriate’’ modality. Might people ‘‘see’’ the spatial layout of an auditory array or—as we investigated—‘‘hear’’ the temporal structure of visual changes? While concentrating on visual sequences consisting of temporally random contrast changes, we noticed a natural tendency to hear the temporal sequencing of these changes as well. Of course, the auditory referencing of visual events is not an entirely new experience; for example, many people engage in subvocalized speech when reading text. However, what we encountered when watching visual sequences had a markedly different flavor: It arose automatically, unintentionally, and Address correspondence to Sharon E. Guttman, Department of Psychology, Vanderbilt University, 111 21st Ave. South, 301 Wilson Hall, Nashville, TN 37203; e-mail: [email protected]. PSYCHOLOGICAL SCIENCE 228 Volume 16—Number 3 Copyright r 2005 American Psychological Society without learning or practice. Most notably, we could not ignore the auditory rhythm implied by the visual changes. In the current article, we present three experiments examining the reality of ‘‘hearing visual rhythms.’’ Specifically, these experiments investigated the idea that rhythm (technically termed temporal structure) portrayed solely by visual input receives automatic, obligatory encoding in the auditory domain. Previous research on modality effects in rhythm processing indicates that auditory rhythmic stimuli produce better shortterm memory than comparable visual stimuli (Glenberg, Mann, Altman, Forman, & Procise, 1989), particularly when musically structured (Glenberg & Jona, 1991). These effects could be attributable to the use of modality-specific temporal codes, which are more reliable in audition than in vision, or to the cost of translating visual stimuli into a different format (either auditory or amodal in nature). Collier and Logan (2001) tested these hypotheses explicitly by having participants match two rhythmic sequences either within or across sensory modalities. Performance in all conditions converged at slower presentation rates, suggesting gradual translation to a universal code. Even at the fastest rate, however, performance in the cross-modal conditions—which contained one (presumably well-encoded) auditory sequence—typically did not exceed performance with two visual sequences. From these findings, Collier and Logan concluded that rapidly presented rhythmic sequences become encoded in a modality-specific manner, thus introducing a cost for comparing rhythms across different modalities. In the experiments we report here, the idea of rapid, obligatory cross-modal encoding is addressed using a paradigm probably more sensitive to underlying representations of temporal structure. In our experiments, observers performed a same/different discrimination task concerning the rhythms of two visual sequences. We assessed temporal representations by measuring the extent to which both task-irrelevant auditory information and task-irrelevant visual information interfered with performance. In interference paradigms, the extent of task disruption reflects the extent to which the representation of the to-be-encoded information overlaps with the representation of the irrelevant information. If visually presented temporal sequences automatically become represented in an auditory manner, then incongruent auditory information should impair processing of the visual stimuli. By contrast, if comparison of two visual temporal sequences utilizes visual representations, then incongruent auditory signals should have minimal effect; however, manipulations that disrupt the visual similarity of the two sequences—even if along a task-irrelevant dimension— should disrupt processing.